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(54) A robot control system 

(57) A robot control system, by means 
of which a robot can be programmed, 
by demonstration, to repeat a job sequ- 
ence, consists of a camera 1, which 
senses a sequence of patterns, during 
the demonstration, which are labelled 
and stored in a job sequence store 5 
and a disc store 6. A scoring unit 4 
compares subsequent in-coming im- 
ages, sensed through the camera 1, 
with stored and labelled images to 
identify a particular required object, and 
generates error signals, which are pas- 
sed to mechanical servos provided on 
an arm 2 of the robot to cause move- 
ment thereof in accordance with the 
stored job sequence. There can be two 
arms and two cameras. 
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The drawings originally filed were informal and the print here reproduced is taken from a later filed formal copy. 

The matter shown in the printed specification between square brackets was furnished after the filing date of the application, the 
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1/1 



2123172 






1 



GB 2 123 172 A 



1 



SPECIRCATION 

A robot control system 

5 This invention relates to a robot control system. 
Robots, with arms capable of being moved to 
specified co-ordinates in accordance with a prog- 
ram, are already known. Robots incorporating pat- 
tern recognition devices are also known. These 

10 devices are capable of recognising unknown pat- 
terns, which may be images of solid objects by 
generating information indicative of the unknown 
pattern for comparison with stored information 
Indicative of known pattems. The comparison yields 

15 a "score" indicating the extent to which the un- 
known pattern resembles each of the known pat- 
tems for which information is stored, and the 
unknown pattem is identified with the known pattern 
which it most resembles, provided that the relevant 

20 score is sufficiently good. 

However, in order to be economical, a robot 
should be able to work at speeds which are at least 
comparable with those attainable by a human 
worker and the pattem recognition techniques em- 

25 ployed heretofore in the field of robotics may not 
operate at a sufHciently high speed to enable full use 
of the robot's capabilities to be made. 

Another problem which may arise is that an 
operator will generally require a substantial know- 

30 ledge of computer techniques before he is capable of 
programming the robot to perform even a simple 
operation. 

It is therefore an object of the present invention to 
provide an improved robot control system which 
35 substantially alleviates at least the afore-mentioned 
problems. 

According to the invention there is provided a 
robot control system, wherein a robot can be 
programmed by demonstration to repeat a job 

40 sequence, comprising a pattem recognition device 
including means for sensing and labelling image and 
position co-ordinates of known objects in various 
attitudes and various manipulations, means for 
storing the sensed and labelled images, means for 

45 searching for and recognising an image of a particu- 
lar required object at a certain stage of the job 
sequence by comparing the image of the required 
object with the sensed and labelled image thereof, 
and means for manipulating the required object to a 

50 desired location in accordance with the job sequence 
by the use of reference positions in a set of chosen 
co-ordinates, in which the robot works during the job 
sequence. 

The robot preferably senses the images shown by 
55 an operatore via a vidicon television camera, or, 
alternatively, a flying spot scanner, using eight 
variables, which have been shown to work in 
simulation therewith. It may be convenient to 
amount the camera on an arm of the robot which is 
60 to perform an operation, or alternatively the camera 
may be fixed. 

The co-ordinates in which the robot works need 
not be expressed in any absolute dimensions, as any 
arbitrary scale that is convenient may be used. For 
65 example, if it were required to insert a bolt into a 



hole, visual recognition and location of the hole or its 
surroundings would provide one set of co-ordinates, 
visual recognition and location of the bolt in an 
artificial hand of the robot would provide a second 
70 set of co-ordinates, and the difference between these 
co-ordinates would provide the necessary data for 
the servo control of the robot 

Three modes of operation, i.e. a search mode, a 
recognition mode and a manipulation mode, are 
75 attainable with the control system in accordance 
with the present invention. 

The preferred type of recognition mode is similar 
to that described in Patent No. 1,234,941, but 
including some modification to provide a substan- 
80 tially more automatic program procedure. The 
search mode and the manipulation mode utilised in 
the present invention are both extensions of the 
above-mentioned patent. 

It is assumed that, in normal operation, objects 
85 which are either components to be included in an 
assembly or things which require some operation as 
part of a manufacturing process will be placed within 
reach of the robot. 

In order to proceed the robot must find the correct 
90 object, in accordance with the job sequence, and 
discern what attitude it is in. 

The recognition logic of Patent No. 1,234,941 can 
recognise an object as seen from one point of view 
and can allow for perspective effects of a single 
95 plane. Thus, in practice, most solid objects could be 
tumed through ±20* away from the original view 
and still be correctiy recognised. However, this is 
only an approximate solution, as the perspective 
allowed for is only that associated with a single 
100 plane. In general, a plane containing imFK)rtant 
features can be selected through the object and the 
distortion of those parts of the object outside the 
plane can be ignored, for recognition purposes, 
provided the total tilt does not exceed, say ±20°. 
1 05 As long as the lengths or diameters of the objects 
are significantly different, it is quite adequate for 
recognition purposes to store four or five images of 
each object at chosen angles to the axis thereof. 

The above-mentioned patent provides a method 
110 which can be arranged to accommodate a ±45** 
rotation about the camera axis, together with all 
perspective effects of the predominant plane of the 
object, apart from small errors at the end thereof. 

Thus, with five stored views of each object, the 
115 present control system can recognise an object in 
any attitude. 

In order to carry out this recognition procedure, 
the robot starts with an image from the camera 
which contains at least a large part of the object, 
1 20 preferably towards the centre of the field. 

At a particular stage in the assembly or fabrication 
process, in accordance with the job sequence, the 
system is required to recognise only one particular 
object, for example a particular type of bolt The 
1 25 recognition system need therefore only classify two 
types of object - the correct type and all wrong types. 

In order to recognise the correct object from all 
others, the system may require images of the others 
for comparison. It is preferable that these should 
130 each be given an indicium of some kind, for 
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example, a name or a number, so that, if the system 
is later required to select a different object, it can 
make use of some of these comparison patterns.' 

An iterative teaching process is used, wherein the 
5 objects are shown, in turn, to the robot until it is able 
to distinguish between each object. 

The television camera used in the present inven- 
tion is provided with two rasters, which scan the 
image in accordance with certain parameters, simi- 
10 lar to those used in Patent No.1,234,941. 

During the search mode, a first raster remains 
locked on to an object being worked on, using error 
signals to hold an image of the part seen in * 
registration. 

15 A second raster is locked on to a tool on the object 
fc>eing added to the assembly. The image of this is 
kept in registration by means of error signals derived 
from the difference between the image of the tool (or 
other object) and a stored image, by moving the arm 

20 of the robot so as to correct the errors. 

During the teaching phase an operator causes the 
second raster to move and be altered so as to view 
those positions to which he wishes the tool to go. 
Once the process has started, small movements 

25 can be achieved simply by altering the scan raster. 
Larger movements may involve changes of attitude 
beyond those which can be simulated by changing 
the raster, or the size of the raster may become 
either too big or too smalt to be convenient. It is also 

30 possible that parts of other objects may ot>scure 
direct vision. To overcome these difficulties the 
system is arranged so that it automatically stores 
new patterns. 
In order to make the error signals produce the 

35 correct arm motions, it is necessary to know the 
approximate positions of the arms and a small 
subsidiary computer is provided to carry out the 
necessary trigonometry. Amplitudes need not be 
very accurate, but it is important to get the correct 

40 sign for each component. 

A pattem that has just been stored should t>e 
substantially identical, apart from random noise, 
with the incoming television signal. 
The operator should place the indicium (e.g. name 

45 or number) of the tool that should be in use on a 
register, so that new stored patterns are always 
labelled and so that if another pattern which is the 
image of some other tool starts to fit better than the 
correct image, then it can be readily detected by 

50 noting disagreement between the labels. 

The parameters associated with each pattem in 
the store are also attached to the pattern. These 
parameters include the relative position of the raster 
concerned to the other raster being used as a 

55 reference position on the object being worked on. 
In the search mode, the first raster is locked on to a 
prominent feature, such as the corner of a tray from 
which components are to be taken. The second 
raster is then made to assume, in sequence, each of 

GO a series of different positions relative to the first 
raster. This series of positions is originally chosen by 
the operator. A scoring mechanism is provided, 
which detects a sufficiently good fit with a stored 
pattem, and then this patter is selected and scanned 

65 in order to produce error signals indicating how to 



improve its matching with the incoming television 
signal. Over a limited range, these error signals are 
allowed to act so as to improve said matching by 
alteration of the selected stored pattern. 

70 From then on, location and identification is 

effected as described in Patent No. 1,234,941, except 
that it is desirable to automate the learning proce- 
dure to a greater degree. This is achieved by 
providing the operator with means of adjusting the 

75 second raster to a suitable size and bringing the 
object to be recognised into view with the register 
loaded with the appropriate indicium. He then 
releases the pattern to move under its own servo 
action. 

80 From the patterns already in the store, the one 
with the highest score provides the error signals for 
position. If the position moves towards the position 
indicated by the operator as the correct one, the 
pattern is allowed to settle. The height and width 
85 servo is now allowed to change the pattem and, if it 
remains within certain tolerances, the parallelogram 
is then allowed to servo. This process is continued 
until all the parameters have been put under servo 
control. If, during this process, the raster starts to 
90 move out of the agreed tolerances for position, size, 
parallelogram distortion, etc., then a new pattern is 
stored. That new pattern is marked with a normalis- 
ing signal, which causes the system to jump to the 
position and size selected by the operator. 
95 If the pattern settles under servo control but has 
the wrong indicium, then another with the correct 
indidum will be stored. 

This routine is now tested by moving the raster 
away from the correct position and checking that it 

1 00 returns under servo control and identifies the correct 
pattem by the highest score, tf rt does not then the 
necessary extra patterns must be inserted. This 
routine can be provided automatically under compu- 
ter control, which moves the raster away from the 

105 correct position by perhaps half its width in all 

directions in order to check for correct servo action 
and identification. 

It can now be appreciated that the present inven- 
tion has a substantial number of advantages, which 

110 include the requirement of any knowledge of com- 
puter programming being unnecessary, because the 
operator can easily programme the robot to perform 
a sequence of movements by merely demonstrating 
the sequence himself. 

115 The present invention can therefore also enable a 
robot to be programmed to do a simple operation in 
about the time ittiakes a man to demonstrate that 
operation and thereafter to repeat it at substantially 
the same speed. 

1 20 Moreover, the control system in accordance with 
the invention can be made from standard integrated 
circuits, so that the system can be sold at a 
reasonable price. 
The invention will now be further described by 

1 25 way of example only with reference to the accom- 
panying drawing, the single figure of which shows a 
block circuit diagram of one embodiment of the 
present invention. 
Referring to the figure, a television camera 1 has a 

130 programmable scan raster and in general the raster 
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has two properties which make it different from a 
normal camera. These properties are the ability to 
scan two areas of the light sensitive screen with 
altemate frames, and a preferable option of scan- 
5 ning each area with lines in two directions at right 
angles, or at least at very different angles. 

A mechanical arm 2 is provided with as many 
types of freedom as are considered necessary for the 
jobs it must do. The arm 2 also has rough position 

10 sensors, which are connected to a computer 3 and 
from which it is possible to compute the relationship 
between a required change in an image and a 
particular movement of individual joints of the arm. 
That is to say it is possible to compute how the arm 2 

15 should move in order to correct any particular 
combination of error signals. 

The system works in three modes of operation, i.e. 
a search mode, a recognition mode and a manipula- 
tion mode, by using stored patterns. An operator 

20 generates control signals, conveyed by a line 8, to 
'determine the original sequence of stored patterns 
since they represent views which are rather like 
frames from a cine-film showing the sequence of 
events during the assembly or fabrication process, 

25 or during successive stages of recognition. However, 
fewer frames are required in the present invention 
than are required in a cine-film, as each frame 
represents an image which can be altered to allow 
for motion, including perspective. 

30 A scoring unit 4 is a special computer which holds 
the patterns in a set of dynamic buffer stores. A 
number of patterns, say 64, are available in these 
stores and parallel scoring devices are at all times 
comparing all the patterns with the incoming frame 

35 from the camera 1 . 

At the same time, logic built into the scoring unit 4 
provides error signals, in the manner described in 
Patent No. 1 234 941, which are either sent to the 
mechanical servos on the arm 2 or to the camera 1, 

40 The computer 3 is needed to change error signals 
from picture co-ordinates to arm space co-ordinates 
when the arm is receiving the error signals. 

The operator originally controls the television 
raster during the teaching phase. The first and last 

45 values for all parameters such as "x" position, "y" 
position, vertical scale, horizontal scale, etc. are 
recorded as part of the label on each pattern, 
together with the name of the object concerned and 
an arbitrary number for the purpose of identifying it 

50 in the store. 

The same pattern may be found to be useful at a 
later stage in the sequence of operations, so it may 
be convenient to provide an additional store (not 
shown in the figure) to which the labels may be 

55 transferred, so that different instructions may be 
associated with any one pattern on different occa- 
sions. Thus, there may be several entries during the 
sequence of operations for each stored pattern. 
When the arm 2 is being controlled by the system 

60 the pattern with the highest score gives the values 
for the raster position and dimensions which are 
then interpolated between the first and the last 
values so as to provide the camera raster with 
smoothly changing parameters. The position is 

65 usually with respect to another raster, this latter 



raster being locked on to a reference pattern holding 
the position of a prominent local feature, which is 
used as a reference point 

Thus during normal servo operations two lots of info- 
70 rmation i.a servo infomnation and parameters defining 
the second raster, go to the camera 1 on altemate 
frames together with relative position of the two rasters. 

The servo information holds the reference pattern 
in position against the image of the reference point 
75 The parameters defining the second raster are 
simply the operator's instruction reproduced, and 
they provide co-ordinate data to mgve the second 
raster in relation to the first raster. Thus, any motion 
of the camera, accidental or otherwise, does not 
80 affect the relative position of the two rasters. 

Servo error signals derived from the second raster 
are passed to the mechanical servos which are thus 
not affected by small movements of the camera. 

This procedure has the advantage that it is largely 
85 self testing since the operator is made aware at once 
if the arm servos fail to respond to their error signals 
as he intends. The system will closely repeat its 
operation as the interpolation can be made to re- 
produce the OF»eratoKs original values very accurately. 
90 A convenient method of passing the information 
to the system so as to alter the raster in the camera 
during the teaching mode is to provide a dummy 
object to represent the tool or the piece which is to 
be assembled. For example, to mount a ball on the 
95 end of lazy tongs, first the lazy tongs may be 
attached to a pillar in front of the operator. By 
holding the ball and moving it in three dimensions 
relative to the pillar the operator can express the 
intention of moving the tool in the corresponding 
100 direction relative to the camera. Change of length of 
the lazy tongs causes the image in the camera to 
change size. Change of direction from side to side 
produces an "x" shift of the image and an up or 
down motion produces a shift A switch enables 
105 the connection to be broken if the operator wishes to 
continue the motion beyond that which can be 
initiated by the lazy tongs. 

Movement of the ball on the lazy tongs need not be 
on the same scale as the motion of the tool. It can be 
1 1 0 helpful to have a "fine control" in which a large move- 
ment of the ball aives a small movenr>ent of the tool. 

The ball itself is rotatable. Two modes are pro- 
vided for imparting rotation to the tool. 

A rotation about an axis perpendicular to its long 
115 axis, if it has one, can be imparted by changing the 
raster in the camera so as to simulate the desired 
movement which is the same method as for "x" and 
V positions and for range. 

If it is necessary to rotate about the axis of a long 
120 thin object, this method is not applicable and an 
alternative mode is provided in which the error 
signals are sent via the computer 3 to the arm 2 
without using the camera 1 for this purpose. During 
this mode of operation the system stores new 
125 patterns and the labels of these record the signals 
which were sent to the servos. However, the process 
needs to be controlled with direct feed back from the 
arm or with tactile feed-back, as visual feed-back is 
not adequate. 
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Job sequence store 5 records the sequence of 
patterns used in the teaching routine. When the 
system is in its manipulation mode the normal 
sequence of pattern involves each pattern using a 

5 different sized raster from the one before. Furth- 
ermore the transfer from one pattem to another uses 
information stored in the job sequence store 5 to 
give the identity of the pattem and its position and 
size. As soon as the transfer is complete the highest 

10 score system is used to check that the correct 

workpiece or tool is in view. A waming is sounded if 
the number on the pattern with the highest score 
differs from the number given by the job sequence 
store 5. 

15 In genera! the highest score determines the next 
pattem to be used. On some occasions, particularly 
in the recognition and search modes, the next 
pattern chosen may not be the next pattern in the 
sequence. This does not generally matter provided 

20 the one chosen is substantially close to it 

On some occasions part of a sequence will be 
repeated in a later sequence on the same job, for 
example, many parts which are all alike may be 
taken from a tray to be put into different places on an 

25 assembly. 

At the end of the repeated sequence the system 
must be prevented from repeating the first sequ- 
ence, which tt would do if the highest score routine 
were followed. This may be carried out by marking 

30 those patterns which must be followed by the 
pattern indicated by the job sequence store 5. This 
mark is deposited when the pattem is used for the 
second time and the sequences diverge. 
A disc store 6 feeds the storing unit 4 with patterns 

35 well in advance of when they are needed. 

During the teaching phase the patterns which are 
first recorded from the camera 1 in the scoring unit 4 
are passed to the disc store 6 so that they can be 
reproduced when needed. The job sequence store 5 

40 indicates which ones are going to be needed. 

The operator can ask for a particular pattern in the 
store 5 to be used as the next one in a sequence by 
entering the number on a keyt>oard (not shown in 
the figure). A monitor 7 displays any pattern called 

45 for by the operator so that he can inspect parts of 
any past routine. 

If a routine is already learnt by the system the 
operator may ask for it as a subroutine by putting the 
first number of that routine on the pattern in use at 

50 the time the subroutine is to be entered. For this to 
work the arm 2 must be in a suitable |x>sition so that 
it starts the routine correctly. 

In certain limited circumstances it is possible to 
program using mainly existing subroutines which 

55 would be called upon by number. 

As a programming facility the operator may ask 
for a different sized raster in a new position at any 
time and with the system in any mode. When this is 
done the information of the identity of the new 

60 pattem to be used, and the size and position of the 
raster to be used with it are recorded in the job 
sequence store 5 at the appropriate place in the 
sequence. 

Once the data from the job sequence store 5 has 
65 been used to determine the pattern and its initial 



position, the scoring system is given power to 
decide if this pattern is to be kept for use in providing 
servo error signals and the servo error signals are 
given control of keeping the image in registration, 
70 either by moving the camera raster within the 
camera or moving the arm with the tool or other 
object on it. 

The camera 1 may be attached to the mechanical 
. arm 2, so as to move with the amn, or altematively, it 
75 may be fixed, as in the present embodiment. 

Preferably there should be two arms and two 
cameras provided on the robot, as many jobs require 
the use of two hands. 

80 CLAIMS (Filed on 17.6.83.) 

1 . A robot control system, wherein a robot can 
be programmed by demonstration to repeat a job 
sequence, comprising a pattem recognition device 

85 including means for sensing and labelling image and 
position co-ordinates of known objects in various * 
attitudes and various manipulations, means for 
storing the sensed and labelled images, means for 
searching for and recognising an image of a particu- 

SO lar required object at a certain stage of the job 
sequence by comparing the image of the required 
object with the sensed and labelled image thereof, 
and means for manipulating the required object to a 
desired location in accordance with the job sequence 

95 by the use of reference positions in a set of chosen 
co-ordinates, in which the robot works during the job 
sequence. 

2. A robot control system as claimed in claim 1 
wherein said means for sensing images comprises a 

100 television camera having a programmable scan 
raster, which can scan first and second areas of a 
light sensitive screen of the camera during alternate 
frames. 

3. A robot control system as claimed in claim 2 
105 wherein said television camera includes means for 

scanning each of said areas with first and second 
lines substantially orthogonal to each other. 

4. A robot control system as claimed in any 
preceding claim wherein said means for storing the 

110 sensed and labelled images includes a job sequence 
store, in which a sequence of patterns, sensed 
during said demonstration, is stored, each of said 
patterns having a label stored therewith. 

5. A robot control system as claimed In any 
115 preceding claim wherein said means for searching 

for and recognising an image of a particular required 
object comprises a scoring unit which generates 
error signals to control movement of the robot, in 
accordance with the job sequence. 

1 20 6. A robot control system as claimed in any 

preceding claim including display means for display- 
ing a selected part or parts of the job sequence. 

7. A robot control system substantially as herein 
described with reference to the accompanying 

125 drawing. 
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